Machine vision method and system

ABSTRACT

An optical system for automated vision and detection. The system includes a light source configured to emit a beam into an environment and a first diffractive optical element. The beam passes through the first diffractive optical element, resulting in a plurality of beams. One or more of the plurality of beams are reflected by the environment, resulting in reflected beams. A second diffractive optical element of the optical system is configured to receive the reflected beams. A detector in alignment with the second diffractive optical element receives the reflected beams. The detector is configured to determine wave data from the reflected beams and generate a plurality of phasorgrams in a single image representing the wave data. The optical system also includes a processor configured to receive the single image and generate a representation of the environment. A control computer is configured to receive the representation of the environment from the processor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 62/652,939, filed on Apr. 5, 2018 and entitled “Machine VisionMethod and System,” the entirety of which is incorporated herein byreference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure is directed generally to system and methods forautomated vision and detection and, more particularly, to an opticalsystem for deriving phase information of an environment.

2. Description of Related Art

Machine vision is a name given to the systems and methods used toprovide automatic imaging-based detection and analysis. These methodsand systems can be utilized for a wide variety of applications,including but not limited to guidance systems, inspection systems,processing systems, sensor systems, and many others.

Machine vision systems can be implemented with a wide variety oftechnologies and methodologies. A system may include, for example, animaging device configured to obtain imaging information about theenvironment within which is it designed to operate. The imaging devicemay be a traditional two-dimensional light camera, and/or it may be amultispectral imager, hyperspectral imager, infrared imager, X-rayimager, and/or other imagers. The system may also include a processor orother processing device configured to receive, store, and/or analyze theimages obtained by the imaging device, which produces an output that maybe utilized in one or more downstream applications.

Just one example of a machine vision system is an automated imagingsystem for a moving vehicle such as a car, truck, and other type ofvehicle, which can be utilized to supplement a driver's control of avehicle, or to supplant the driver's control in the case of autonomousor semi-autonomous vehicles.

One common machine vision system utilized in moving vehicles is LIDAR,an imaging method that illuminates an environment with potentialtargets/objects with pulsed laser light, and measures distances topotential targets/objects based on reflected pulses as detected by asensor. A three-dimensional representation of the environment can thenbe digitally created using the measured distances.

However, LIDAR and similar systems suffer from many deficiencies thatconstrain their ability to operate as efficient and highly-effectiveimagers for machine vision systems. Among other limitations, LIDARsystems are expensive and suffer from low resolution, and thus aretypically unable to create the high-resolution three-dimensional maps ofthe environment necessary for imaging devices operating in high speedsituations, and/or necessary for imaging devices operating inenvironments requiring high-resolution three-dimensional maps.

Accordingly, there is a continued need for affordable and efficientoptical systems capable of creating high-resolution three-dimensionalmaps of an environment.

SUMMARY OF THE INVENTION

The present disclosure is directed to inventive methods and systems forautomated vision and detection. According to an embodiment, the presentinvention is an optical system. The optical system includes a lightsource configured to emit a beam into an environment and a firstdiffractive optical element. The beam passes through the firstdiffractive optical element, resulting in a plurality of beams. One ormore of the plurality of beams are reflected by the environment,resulting in reflected beams. A second diffractive optical element ofthe optical system is configured to receive the reflected beams. Adetector in alignment with the second diffractive optical elementreceives the reflected beams. The detector is configured to determinewave data from the reflected beams and generate a plurality ofphasorgrams in a single image representing the wave data. The opticalsystem also includes a processor configured to receive the single imageand generate a representation of the environment. A control computer isconfigured to receive the representation of the environment from theprocessor.

According to another embodiment, the present invention is a system forautomated vision and detection. The system includes an imager configuredto obtain a plurality of phasorgrams of an environment and each of thephasorgrams comprises phase information. The imager is configured togenerate a single image using the phase information. The system alsoincludes a processor configured to derive a representation of theenvironment based on the single image and an implementation moduleconfigured to receive the representation of the environment and generatea response based thereon.

According to yet another embodiment, the present invention is an opticalsystem for generating digital images. The system includes a light sourceconfigured to emit a beam into an environment and a first diffractiveoptical element. The beam passes through the first diffractive opticalelement, resulting in an illumination wave. The illumination wave isemitted into the environment and is reflected by the environment,resulting in a reflected illumination wave. A second diffractive opticalelement of the optical system is configured to receive the reflectedillumination wave. A detector in alignment with the second diffractiveoptical element receives the reflected illumination wave. The detectoris configured to determine wave data from the reflected illuminationwave and generate an image representing the wave data. The opticalsystem also includes a processor configured to receive the image andgenerate a representation of the environment.

According to an embodiment, the methods described or envisioned in U.S.Pat. No. 8,040,595 are directed to a method and/or system forreconstructing a wave, including interpolation and extrapolation of thephase and amplitude distributions. For example, according to one aspectis a method for reconstructing a wave. The method includes the steps of:(i) illuminating a specimen at an output plane to provide an output wavecomprising specimen information, the output plane including a nullregion in which the output wave has a value of zero; (ii) applying anumber N of different phase filters of known phase shift to the outputwave at or near the output plane to create N phase-shifted waves eachhaving a phase shift that corresponds to a respective phase filter;(iii) measuring at a diffraction plane only a portion of the amplitudedistribution for each of N diffraction patterns to provide measured andunmeasured portions of the amplitude distributions of the diffractionpatterns, each diffraction pattern corresponding to a respective one ofthe phase-shifted waves; (iv) inverse Fourier-transforming each of themeasured portions of the amplitude distributions of the diffractionpatterns to produce a respective computed estimate of the phase-shiftedwave at the output plane, the computed estimates comprising bothamplitude and phase information; (v) applying a respective inverse phasefilter to each of the computed estimates of the phase-shifted waves toremove the phase-shift introduced by the corresponding phase filter toprovide computed estimates of non-phase-shifted waves at the outputplane; (vi) correcting the computed estimates of the non-phase-shiftedwaves by setting the values of the amplitude and phase to zero for thoseportions of the computed estimates of the non-phase-shifted waves whichcorrespond to the null region of the output plane to provide a pluralityof corrected estimates; and (vii) recovering the amplitude and phase inthe diffraction plane of the unmeasured portion of the amplitudedistributions of the diffraction patterns based on the correctedestimates, whereby the amplitude and phase information of the unmeasuredportion is recovered to achieve super-resolution.

According to another aspect is a method for reconstructing a wave. Themethod includes the steps of: (i) illuminating a specimen at an outputplane to provide an output wave comprising specimen information, theoutput plane including a null region in which the output wave has avalue of zero; (ii) applying a number N of different filters to theoutput wave at or near the output plane to create N filtered waves eachcorresponding to a respective filter; (iii) measuring at a diffractionplane only a portion of the amplitude distribution for each of Ndiffraction patterns to provide measured portions of the amplitudedistributions of the diffraction patterns, each diffraction patterncorresponding to a respective one of the filtered waves; (iv) inverseFourier-transforming each of the measured portions of the amplitudedistributions of the diffraction patterns to produce a respectivecomputed estimate of the filtered wave at the output plane, the computedestimates comprising both amplitude and phase information; (v) applyinga respective inverse filter to each of the computed estimates of thefiltered waves to provide computed estimates of non-filtered waves atthe output plane; (vi) correcting the computed estimates of thenon-filtered waves by setting the values of the amplitude and phase tozero for those portions of the computed estimates of the non-filteredwaves which correspond to the null region of the output plane to providea plurality of corrected estimates; and (vii) recovering the amplitudeand phase in the diffraction plane of the unmeasured portion of theamplitude distributions of the diffraction patterns based on thecorrected estimates, whereby the amplitude and phase information of theunmeasured portion is recovered to achieve super-resolution.

According to an embodiment, the automated vision and detection systemcan be a component of or integrated into any system or device requiringautomated vision or detection. The automated vision and detection systemcan be a component of or otherwise integrated into a guidance system,inspection system, processing system, sensor system, and/or many othersystems. As just one embodiment, the automated vision and detectionsystem can be a component of or otherwise integrated into an automatedimaging system for a moving vehicle such as a car, truck, and other typeof vehicle. As another example, the automated vision and detectionsystem can be a component of or otherwise integrated into a handhelddevice configured to obtain imaging information from or about theenvironment.

As another example, the automated vision and detection system can be acomponent of or integrated into a system configured to be utilized todetect, map, or otherwise characterize air or wind patterns. Forexample, the automated vision and detection system can be utilized tomake a three-dimensional map of air, wind patterns, wind shear, and/orother air structures.

As another example, the automated vision and detection system can be acomponent of or integrated into a system configured to be utilized todetect, map, or otherwise characterize water or wave patterns. Forexample, the automated vision and detection system can be utilized tomake a three-dimensional map of water, water or wave patterns, waterflow, and/or other water structures. Many other types of vision ordetection systems are possible.

These and other aspects of the invention will be apparent from theembodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following descriptiontaken in conjunction with the accompanying drawings in which:

FIG. 1 is a diagrammatic representation of an optical system forautomated vision and detection, in accordance with an embodiment;

FIG. 2 is a diagrammatic representation of the optical system in use ina vehicle, in accordance with an embodiment;

FIG. 3 is a schematic representation of a machine vision method, inaccordance with an embodiment; and

FIG. 4 is a schematic representation of a machine vision system, inaccordance with an embodiment.

FIG. 5 is a diagrammatic representation of an optical system forautomated vision and detection, in accordance with an alternativeembodiment;

FIG. 6 is a diagrammatic representation of an optical system for acontrolled environment at close range, in accordance with an alternativeembodiment; and

FIG. 7 is a prior art flow chart of the algorithm used at the processorof the optical system, in accordance with an embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Aspects of the present invention and certain features, advantages, anddetails thereof, are explained more fully below with reference to thenon-limiting examples illustrated in the accompanying drawings.Descriptions of well-known structures are omitted so as not tounnecessarily obscure the invention in detail. It should be understood,however, that the detailed description and the specific non-limitingexamples, while indicating aspects of the invention, are given by way ofillustration only, and are not by way of limitation. Varioussubstitutions, modifications, additions, and/or arrangements, within thespirit and/or scope of the underlying inventive concepts will beapparent to those skilled in the art from this disclosure.

Referring now to the figures, wherein like reference numerals refer tolike parts throughout, FIG. 1 shows a diagrammatic representation of anoptical system 10 for automated vision and detection. The optical system10 in FIG. 1 can be used in uncontrolled environments, such as roadways,and at a distance (e.g., >20 ft.). The optical system 10 comprises alaser 12 positioned in line or at a first diffractive optical element14. In the depicted embodiment, the first diffractive optical element isa computer-generated hologram (CGH) device. As shown in FIG. 1, thelaser 12 emits a beam 16 at the first diffractive optical element 14.The first diffractive optical element 14 is a beam splitter and splitsthe incoming beam 16 into multiple beams 18 (or a plurality of beams),which exit the first diffractive optical element 14.

In an alternative embodiment, shown in FIG. 5, the laser 12 is replacedwith a VCSEL laser array as a light source. The VCSEL laser array 12effectively multiplies the number of beams 18 in the field of view bythe number of VCSELs when used with the first diffractive opticalelement 14. In the depicted embodiment, using a VCSEL laser array 12quadruples the number of beams 18 in the field of view. However, intheory, the VCSEL laser array 12 can be configured to generate anydesired number of beams 18. For example, if a VCSEL “chip” (laser array)12 with 16 beams 18 is used and the first diffractive optical element 14has a 200-beam fanout, 16*200=3200 beams will be produced at once.Practically, the power of the VCSEL laser array 12 will be limited to apractical fanout number.

The multiple beams 18 exit the first diffractive optical element 14 andcontinue into a surrounding environment 19 until they contact an object20, such as that shown in FIG. 2. As an example, the environment 19 maybe a factory, an assembly line, a warehouse, the outdoors, a road, orany other type of environment. If one or more of the multiple beams 18contacts the object 20, those beams are reflected back toward theoptical system 10 as reflected beams 22. As shown in FIG. 1, thereflected beams 22 pass through a second diffractive optical element 24.The second diffractive optical element 24 may also be a CGH device. Inthe depicted embodiment, the second diffractive optical element 24 mayadditionally include a lens (or system of lenses) 24′. Similarly, thefirst diffractive optical element 14 may also comprise a lens 24′ (orsystem of lenses). Further, one lens can be used for both diffractiveoptical elements 14, 24 if a beam splitter is used—as understood by aperson of ordinary skill in the art).

The second diffractive optical element 24 enables the formation ofmultiple phasorgrams 28. In the depicted embodiment, a plurality ofphasorgrams 28 are produced simultaneously. The second diffractiveoptical element 24 has filters which allow for the creation of multiplephasograms 28, a phasogram 28 for each point in the environment 19 thatis reflected (in the reflected beams 22).

Still referring to FIG. 1, the optical system 10 additionally includes adetector 26, such as a charge-coupled detector (CCD) device (e.g.,camera). The detector 26 is aligned with the second diffractive opticalelement 24 or positioned relative to the second diffractive opticalelement 24 such that the multiple phasorgrams 28 exiting the seconddiffractive optical element 24 are received at the detector 26. Thephasorgrams 28 (including wave data) are acquired simultaneously at thedetector 26 for simultaneous processing, rather than a sequentialacquisition, like traditional optical systems/methods, such as LIDAR.The detector 26 obtains and extracts wave information (or “wave data”)from the phasorgrams 28, interprets the wave data, and produces a singleimage 30.

As shown in FIG. 1, the optical system 10 further comprises a processor32. The processor 32 is connected to the detector 26 via a wired orwireless connection. The processor 32 is capable of performing digitalsignal-processing functions, such as two-dimensional Fast FourierTransforms. In an embodiment, the processor 32 is a GPU processor, whichpermits the parallel processing of simultaneous information, increasingthe speed of processing. The single image 30 is transmitted (via a wiredor wireless connection) from the detector 26 to the processor 32. Theprocessor 32 performs various steps of data processing using the singleimage 30 to create a single representation 34 of the environment 19(FIGS. 1-2), including any objects 20 in the environment 19. Therepresentation 34 of the environment is then transmitted (via a wired orwireless connection) from the processor 32 to a control computer 36. Thecontrol computer 36 can be an existing computer in the vehicle, acomputer within the optical system 10, or a remote computer. The controlcomputer 36 is programmed with various instructions or “rules” forgenerating a response to the representation 34.

Turning now to FIG. 3, there is shown a schematic representation of amethod 100 for automated vision and detection. At step 110 of themethod, the optical system 10, as described above, is provided within avehicle. As described herein, a vehicle can be any movable device,including an automobile, a plane, a drone, etc. The embodiment of theoptical system 10 described herein is installed within an automobile, asshown in FIG. 2, although the optical system 10 can be scaled andapplied to other movable devices. In this embodiment, the optical system10 is installed within an automobile such that the processor 32 of theoptical system 10 is connected to the control computer 36 of theautomobile.

An exemplary embodiment of the optical system 10 integrated within anautomobile is shown in FIG. 4. In FIG. 4, the optical system (alsoreferred to as “the automated vision and detection system”) 200 includesan imager 210. The imager 210 comprises the laser 12, the first andsecond diffractive optical elements 14, 16, and the detector 26components of the optical system 10 of FIG. 1. The optical system 200 inFIG. 4 also includes a processor 220 (analogous to the processor 32 inFIG. 1). The processor 220 is in communication with the imager 210, andis configured to receive one or more images 30 from the imager 210 forprocessing. As described or otherwise envisioned herein, the processingby processor 220 can comprise any analysis or processing required toimplement the goals or design of the automated vision and detectionsystem 200. In particular, the processor 220 may process the receivedsingle image 30 to derive a representation 34 of the imaged environment19.

The optical system 200 for an automobile in FIG. 4 additionally includesa guidance module 230 (e.g., at the control computer 36 of theautomobile (FIG. 1)). The guidance module 230 or any other module orcomponent is configured to receive processed imaging information fromprocessor 220, such as the generated representation 34 (FIG. 1) of theimaged environment 19 (FIG. 1), among other processed imaginginformation. Accordingly, the module 230 may be any implementationmodule that uses or implements the processed imaging information fromthe processor 220. For example, the implementation module 230 may be aguidance module which utilizes the processed imaging information fromthe processor 220 to guide a vehicle, robot, or other device.

Referring back to FIG. 3, at step 120 of the method, the imager 210obtains one or more phasorgrams 28 of the environment 19 (FIG. 1) inwhich the imager 210 or optical system 200 is located. The environment19 can be any location or area that can be imaged or otherwise assessedutilizing the imager 210, such as the road. The imager 210 obtains oneor more images (i.e., phasorgrams 28) of the environment 19.Specifically, the beam splitting properties of the second diffractiveoptical element 16 of the imager 210 allow for numerous separatephasorgrams 28 to be taken on the same plane simultaneously. Thus,instead of a series of images (or phasorgrams) 28 taken sequentially, asis done in traditional optical systems, the imager 210 described hereintakes all the images 28 at the same time.

Simultaneous detection/processing of the images (phasorgrams) 28 is acrucial difference (as compared to traditional optical systems) forallowing the optical system 10 described herein to work in real-time. Italso allows the optical system 10 to work in a solid state mannerwithout any moving parts. This is a significant improvement overtraditional optical systems because it increases the speed at which theoptical system 200 (and optical system 10) processes the environment 19(FIG. 1). Significantly improving the processing speed of thephasorgrams 28 completely or almost entirely eliminates time distortion.The system 200 may obtain the one or more phasorgrams 28 automatically(such as continuously or pursuant to a timer or other periodic system),and/or may obtain imaging information in response to an internal orexternal stimulus, and/or may obtain imaging information in response toa command from a user or other command provider. As described withreference to FIG. 1, the imager 210 (via the detector 26), generates asingle image 30 from the phasorgrams 28.

At step 130 of the method, the processor 220 processes or otherwiseanalyzes the single image 30 obtained by the imager 210 and provided tothe processor 220. The optical system 200 may provide the single image30 to the processor 220 immediately, and/or in response to a request forthe single image 30 by the processor 220. The single image 30 may alsobe temporarily or permanently stored before and/or after analysis by theprocessor 220. For example, the single image 30 may be stored in adatabase for batch processing, delayed processing, or periodicprocessing. Processing may also be performed in real-time.

The processor 220 analyzes the single image 30 obtained by the imager210 using any method of analysis, including but not limited to one ormore aspects of the methods set forth in detail in U.S. Pat. No.8,040,595. Accordingly, the disclosure of U.S. Pat. No. 8,040,595, whichwas filed as U.S. Pat. App. No. 12/376,890 on Feb. 9, 2009 (claimingpriority to PCT App. No. PCT/US2007/018008 filed on Nov. 2, 2007) andpublished as U.S. Pat. Pub. No. 2001/0032586 on Feb. 10, 2011, entitled“Light Microscope with Novel Digital Method to AchieveSuper-Resolution,” is hereby incorporated herein in its entirety.According to one embodiment, the output of processing or analysis by theprocessor 220 is a two-dimensional or three-dimensional representation34 of the environment 19 in which the single image 30 was obtained.

The '595 patent mentioned above describes how a scalar wave front may berepresented as a two-dimensional complex function by inferring the phasedistribution from the measured amplitude distribution of the wave front.Using the paraxial approximation and the restrictions of Fresneldiffraction, the initial wave front at z=z_(i) which is propagated alongthe z axis (normal to the wave front) is related to the observed wavefront downstream of at z_(i), at z=z_(o), by the equation:

$\begin{matrix}{{U\left( {x_{0},y_{0}} \right)} = {\frac{1}{j\; z\; \lambda}{\exp \left( {j\; {kz}} \right)}{\exp \left\lbrack \frac{\left( {j\; k} \right)\left( {x_{o}^{2} + y_{o}^{2}} \right)}{2\; z} \right\rbrack}{\int{\int{{U\left( {x_{i},y_{i}} \right)}{\exp \left\lbrack \frac{\left( {j\; k} \right)\left( {x_{i}^{2} + y_{i}^{2}} \right)}{2\; z} \right\rbrack} \times {\exp \left\lbrack \frac{\left( {- {j2\pi}} \right)\left( {{x_{0}x_{i}},{y_{0}y_{i}}} \right)}{z\; \lambda} \right\rbrack}{x_{i}}{y_{i}}}}}}} & (1)\end{matrix}$

where U(x, y) is the total complex wavefunction in the (x, y) planenormal to the z axis, z is the drift distance between the initial wavefront and the observed wave front (i.e. z=z_(o)−z_(i)), λ is thewavelength, the subscript i indicates quantities in the initial waveplane, the subscript o indicates quantities in the observed wave planeand k is the free-space wavenumber in radians per wavelength.

An eminently readable derivation of this diffraction equation (1), andthe paraxial and Fresnel diffraction constraints is known. Equation (1),assuming full knowledge of a scalar wave front (a two-dimensionalcomplex function of Cartesian coordinates x and y in a plane normal tothe z direction of propagation), allows for the calculation of thetwo-dimensional complex wavefunction at any drift distance downstream.Based on equation (1), given the amplitude distribution of the complexwave front in the observation plane, one can generate the phase functionof the wave front and thereby provide the information to reconstruct theactual wavefunction in the initial plane uniquely.

To solve the phase retrieval problem, the algorithmic process is cyclic.The complex function representing each phasorgram in the image plane isestimated and then these estimates are used to generate a single newestimate of the complex wavefunction in the initial plane. This newinitial plane complex function is then used to generate better estimatesof the image plane phasorgrams and so on. An index of the goodness ofthese estimates is the squared difference between the estimated andmeasured amplitude of each pixel summed over all the phasorgrams.Dividing this number by the sum of the squared measured amplitudes ofeach pixel over all pixels over all phasorgrams (the definedphasorgrams' energy) gives a normalized index called the fractionalerror. Note that, the smaller the fractional error, the more thephasorgram estimates look like the measured phasorgrams and presumablythe better the complex function estimate resembles the sought function.

FIG. 7 shows a flow chart of the algorithm used at the processor 220(analogous to the processor 32 in FIG. 1). Its process can be followedby beginning with the N phasorgrams, which are amplitude distributionsof complex wavefunctions. One seeks the unique phase distribution whichis valid for each of them. To begin, assume that there is no reason toestimate one possible phase distribution over another. Set the phasedistribution for all the phasorgrams to zero. Therefore, the firstestimate is that they are all real functions. Now, Fourier transformeach of them and retain the N complex transform functions. If theperturbing device associated with a particular phasorgram was a stop,then all the pixels on each transform that were stopped have theirphasors set to zero amplitude. If the perturbing devices were opticallenses or kinoform lenses, then remove the phase effect of theperturber. That is, if the perturber advanced a pixel phasor by 1 rad,then retard that pixel phasor by 1 rad. Carry out the same removalprocedure for all the pixels in all the transform functions.

Now, still referring to FIG. 7, on a per pixel basis, add togethervectorially the phasors for the same pixel from each of the N transformcomplex functions (initial wave-front estimates) and average them.Phasors whose amplitude is zero because they were blocked are notconsidered in the average. Thus, if there were 10 transforms and for aparticular pixel, in three of the transforms, that pixel was blocked,then the sum of the remaining seven phasors is divided by seven. Thisrule does not apply if the perturbers are lenses or kinoforms. There,zero-amplitude phasors are counted as values in the averaging. Theresulting single averaged transform is the new estimate of the initialplanewave front.

Next, apply to this wave estimate the effect of each different perturberin turn. Then, inverse Fourier transform each of the filtered initialwave estimates and retain them. Each of these new complex functions isan uncorrected estimate of the phasorgram of its respective filter orperturber. Correct this estimate by setting the amplitude of each pixelphasor in each of the new phasorgram estimates to the measured amplitudevalue while retaining the phase of the estimate. Now, the next iterationcycle begins with these new phasorgram estimates.

Note what has been done: there were N phasorgrams and complete knowledgeof the N perturbers which created them. No information was in hand aboutperhaps the limited support of the initial wavefunction or about whetherit was a complex function or not and so on. Instead, this algorithm inFIG. 7 relied on the principle of error energy reduction exclusively. Asfor error energy and how it is used, the algorithm begins by estimatingthe complex function in the plane of observation for each of the Nphasorgrams. The phasor for each pixel in all N phasorgrams will haveits amplitude set to the measured amplitude and for convenience thephase of each phasor in set to zero. Each of the N phasorgrams isFourier transformed and the effects of their corresponding perturbersare removed. This yields N different estimates of the complex initialwave. These N estimates must now be combined to yield a single initialwave estimate according to some principle: the error reductionprinciple. Each initial wave pixel estimate has N or fewer phasorestimates and each of these will be changed to the same phasor estimatefor that pixel. That change will require the vector addition to each ofthese phasor estimates of a different vector to change it to the singlefinal phasor estimate for that pixel.

The final phasor estimate should be selected in such a way that the sumof the square of the norm of each of the distinct vectors, which areadded to each of the N or fewer phasor estimates to yield the finalsingle phasor estimate for the pixel, is a minimum. The sum of thisnumber taken over all the pixels in each of the N initial wave estimateswill be called the correction energy. One wants the correction energy tobe as small as possible.

To achieve minimum correction energy achieved, consider a single pixelwith its say L phasor estimates. Let E be the total correction energycontribution of this pixel. Then one has the following equation:

$\begin{matrix}{{E = {{\sum\limits_{i = {1\mspace{14mu} {to}\mspace{14mu} L}}\left( {U_{i} - U_{0}} \right)^{2}} + \left( {V_{i} - V_{0}} \right)^{2}}},} & (2)\end{matrix}$

where (U_(i), V_(i)) is the ith phasor estimate's real and imaginarycoordinates and (U_(o), V_(o)) is the final phasor estimate's real andimaginary coordinates. Expanding yields

$\begin{matrix}{E = {{\sum\limits_{i = {1\mspace{14mu} {to}\mspace{14mu} L}}U_{i}^{2}} - {2U_{i}U_{0}} + U_{0}^{2} + V_{i}^{2} - {2V_{i}V_{0}} + {V_{0}^{2}.}}} & (3)\end{matrix}$

U and V are independent variables; so solving for each separately yields

$\begin{matrix}{\frac{E}{U_{0}} = {{{\sum\limits_{i = {1\mspace{14mu} {to}\mspace{14mu} L}}{{- 2}U_{i}}} + {2U_{0}}} = {0\mspace{14mu} {or}}}} & (4) \\{{{\frac{1}{L}{\sum\limits_{i = {1\mspace{14mu} {to}\mspace{14mu} L}}U_{i}}} = U_{0}};} & (5)\end{matrix}$

similarly

$\begin{matrix}{{\frac{1}{L}{\sum\limits_{i = {1\mspace{14mu} {to}\mspace{14mu} L}}V_{i}}} = {V_{0}.}} & (6)\end{matrix}$

That is, averaging the phasor estimates provides the final phasorestimate that was sought. This procedure only guarantees an extreme butthe physical situation is such that there can be no maximum.

Now, in turn each of the perturbers are applied to this estimate andinverse Fourier transformed to yield N new estimates of the Nphasorgrams in the observation plane. The perturbers (either opticallenses or kinoform lenses or holed stops) have been chosen to maintainthe energy (norm squared) of both the initial uncorrected waveformestimates and the correction functions that were added to them. Thelinearity of the Fourier transform ensures that the new estimates of thecomplex phasorgrams will be composed of two parts; the previousiteration estimate plus the transform of the respective correctionfunctions. Furthermore, Parseval's theorem insures that the energy ofeach of these two parts of a phasorgram estimate will have the sameenergy as it did in the initial plane. Simply put, Parseval's theoremstates that the energies (the integrated squared norm of a complexfunction) of a function in both of the two conjugate Fourier planes willbe the same. One now chooses to make the correction to each pixel phasorin the observation plane which is the smallest possible.

The total error energy over all phasorgrams as the process proceedsthrough the Fourier transforms and the removal of the perturber effectsto the N new estimates of the initial wave front is the upper limit ofthe total correction energy. That is because the new estimate of theinitial wave is the same as the previous estimate of the initial waveplus the transform of the error function that was added in theobservation plane. Parseval's theorem ensures that that error functionenergy is the same in both conjugate planes. Thus, at this point, thealgorithm could return to the previous initial wave estimate, causingthe correction function energy to equal the error energy and thealgorithmic process would stagnate. Neither the correction energy northe error energy would change. However, by averaging the phasorestimates, one is guaranteed to have the lowest total correction energypossible. In this way, the algorithm proceeds from one conjugate planeto the other reducing the correction energy in the initial plane andthen reducing the error energy in the observation plane. The goal, ofcourse, is to find a phase distribution which makes the error energy orthe correction energy zero or as close to zero as possible.

The new iterative algorithm described below is used to solve the phaseretrieval problem (and used by the processors 220, 32 described hereinto process the single image 30 (from the phasorgrams 28)). The definederror energy of the algorithm must decrease with each iteration or atworst remain the same. For the algorithm to work with the embodiment ofthe optical system 10 shown in FIG. 1 (and in FIGS. 5-6), the lightsource (e.g., laser 12) is immediately upstream or downstream of theenvironment 19 and the wave front phase will be changed by a phasefilter (e.g., first and second diffractive optical elements 14, 24) sothat the wave front at an object 20 in the environment 19 will be thesum of the unknown object 20 amplitude and phase distribution and theknown phase filter distribution. The filter phase distribution is known.The filter amplitude distribution is equal to a constant of one. Theoutput wave (at the specimen) will propagate along the Z axis to thediffraction plane located at Z=Z_(d). There, the intensity or rather theamplitude distribution of the wave front will be measured for eachdifferent phase filter successively. The number of diffraction patternsin hand will equal the number (say, N) of phase filters used. Thediffraction patterns will be known.

Computer processing all these wave data is done in an iterative manner.Successive cycles of the algorithm produce better and better estimatesof the object 20 amplitude and phase distributions. There is thepossibility that successive estimates do not change. In this case,additional filters (e.g., first and second diffractive optical elements14, 24) will be required to generate additional diffraction patterns.However, the algorithm is guaranteed not to diverge from the correctestimate in a mean squared error sense.

The wave function in the diffraction plane is the Fourier transform ofthe filtered wave function in the object 20 plane. For no particularreason, begin the first iterative algorithm cycle in the diffractionplane corresponding to one particular filter. With the amplitudedistribution of the wave which was measured, combine it with the bestapproximation for the phase distribution to yield the first estimate ofthe complete wave function for that particular filter in the diffractionplane. Put this estimate through an inverse Fourier transform to yieldan estimate of the filtered specimen wave. In the computer (e.g.,processor 220, 32), use an inverse phase filter to cancel the effect ofthe actual filter. (If an element of the physical filter shifted thephase by say plus 37 degrees, the inverse filter would shift the phaseof that element by minus 37 degrees). This yields the first raw estimateof the object 20 phase and amplitude distribution. Save this object 20estimate. Generally, use two two-dimensional matrices each with X & Yindices which cover the output plane. One matrix contains the Real partof the complex numbers which define the specimen wave function at eachpoint on the object 20 and the other part contains the Imaginary part.Now, do the same procedure with each diffraction plane amplitudedistribution adding the Real and Imaginary parts of the wave formsgenerated into the 2 corresponding Real and Imaginary matrices. Now,divide each number in the 2 matrices by the number of diffractionpatterns (N) which have been used. Also, since we know that the value ofthe true object 20 wave is zero, we can set all values of elementsoutside the object 20 to zero. At this point, we may be able toincorporate any data that we know about the true object 20 wave into theestimated wave function that is contained in the two matrices, alwaystaking care to make the correction as small as possible if there is arange of correction that will satisfy the known a priori constraint.Clearly, at this point our two matrices hold the first estimate of thewave function in the output or object 20 plane. Note that we have takensome number (say N) of recorded diffraction patterns in the diffractionplane to generate just one estimate of the wave function in the outputplane before we apply any phase or occluding filters to it. The nextstep in the algorithm is to generate estimates of the N diffractionpatterns that this estimate of the specimen wave function will produceafter it has been modified by a phase (or occluding) filter.

Take one of the phase filters (e.g., first or second diffractive opticalelements 12, 24) and essentially apply it to the estimate of the object20 wave function in the output plane. Then, propagate the wave to thediffraction plane. In the computer (e.g., processor 220, 32), this isdone by mathematically Fourier transforming the filtered estimate of theobject 20 wave function. The diffraction pattern amplitude distributiongenerated will not match that which was physically measured andcorresponded to that filter. So, replace the diffraction wave amplitudedistribution with the measured distribution leaving all points on thewave at which I have no a priori measured data untouched. As thealgorithm proceeds, these points will assume the value that they musthave. The points so developed may be said to be extrapolated orinterpolated depending on their location in the diffracted wave. Notethat I have not modified the phase distribution of the diffracted wave.Thus, the second estimate of the diffracted wave corresponding to thephase filter chosen is in hand and this diffracted wave function issaved. Do the same procedure for the next N−1 remaining filtered outputwaves yielding a total of N diffracted wave second estimates. These arethe new N diffracted wave estimates with which to begin the next cycleof the iterating algorithm.

A figure of merit is the sum of the squares of the differences betweenthe diffraction amplitude distributions measured minus those estimated.This is the error energy for any particular cycle. It will be found thatthis error energy cannot increase and that given sufficient numbers offiltering data, will always decrease approaching a limit of zero. Ofcourse, zero error will mean that not only has the phase inverse problembeen solved but so too has the extrapolation and interpolation problemsbeen solved in cases where this was required.

The method just described lends itself to implementation in a number ofphysical embodiments, which in turn suggest some possibly usefulvariations of the method. The above-described method may be applied tothe operation of optical system 10 in FIGS. 1, 5 and 6, therebyachieving super-resolution. In order to do this, the reflected beams 22generate the N diffraction patterns at detector 26 required as inputs tothe algorithm. The algorithm is run as described above, generating notonly the phase distribution at the diffraction plane, but also theamplitude and phase distribution at points not measured by detector 26,for example at points beyond the physical extent of detector 26(extrapolation), and/or at omitted or doubtful points in between thosemeasured by detector 26 (interpolation). By using the diffractiveoptical elements 12, 24 (in FIGS. 1, 5 and 6) in conjunction with thealgorithm, diffraction patterns can be inverse-Fourier-transformed inparallel and a considerable speed-up in the operation of the algorithmcan be achieved.

At step 140 of the method, a system, such as the guidance module 230(FIG. 4) or the control computer 36 (FIG. 1) utilizes the representation34 (i.e., processed single image 30) of the environment 19. Any systemcan utilize the output for any goal of the particular system. Forexample, according to one embodiment, the output of processing oranalysis by the processor 220 is utilized by the automated vision anddetection system 200 to achieve one or more goals of the system. In theexample of an automated or semi-automated vehicle, the vehicular systemmay utilize the output to provide information or feedback to a driver.The vehicular system may utilize the information to navigate and controlthe vehicle. As the imager 210 entirely or almost entirely eliminatestime distortion effects on the representation 34 of the environment 19,the vehicle (e.g., via the guidance module 230) has additional time torespond to objects 20 in the environment 19. For example, if a firstvehicle is traveling in a first direction at 70 mph and a second vehicleis traveling in an opposing second direction at 70 mph toward to thefirst vehicle, there is approximately 3 seconds from detection of theother vehicle to contact. Most of the 3 seconds is needed for themechanical response of the vehicle. Thus, by improving the detection(i.e., image processing) time, the vehicle has additional time tomechanically respond. This allows for improve object detection accuracyand an increased safety factor for the vehicle with the above-describedautomated vision and detection system 200.

The automated vision and detection system 200 can be a component of orintegrated into any system or device requiring automated vision ordetection. The automated vision and detection system 200 can be acomponent of or otherwise integrated into a guidance system, inspectionsystem, processing system, sensor system, and/or many other systems. Inanother example, the automated vision and detection system 200 can be acomponent of or otherwise integrated into a handheld device configuredto obtain imaging information from or about the environment. Many othertypes of vision or detection systems are possible.

In an example of an assembly line or other factory or similar setting,the automated vision and detection system 200 may utilize the output toprovide information or feedback to a robot or other automated componentwhich is configured to move or direct movement within the environmentand therefore must have a representation of the environment, includingbut not limited to a real-time or near real-time representation.

In an alternative embodiment, the optical system 10 is used in acontrolled environment, a room in a house (e.g., no rain, fog, or otherweather), and within close range (e.g., ≤20 ft). In one such embodiment,the optical system 10 is integrated with gaming technology. For example,the optical system 10 in FIG. 1 can be integrated or otherwise installedwith a gaming device, such as a gaming console or a gaming mobiledevice.

Traditional gaming technology relies on the projection of a line/gridpattern directed at an environment. The lines/grid are distorted whendeflected off an object in the environment and using known fringeprojection technique, information (data) is directed from the distortedlines/grid collected at a detector. The information is used to create animage, which is processed with signal processing. The signals are thenused by programmable instructions or rules to control the game. Inanother example, mobile devices, such as cellular phones or smartphones,project an array of dots and detect/process the distortion of the arrayreceived at a detector.

However, in both examples, the traditional gaming systems and mobiledevices are limited by the information provided by the line/grid patternand the array of dots, respectively. The optical systems used intraditional gaming systems can only receive and process information fromthe line/grid or patterns and dot arrays. Thus, any information aboutthe environment between the lines or dots is not retrieved or processed.Therefore, the line/grid pattern and dot arrays provide much lessinformation than information regarding the entire wave, which isprovided by the optical system 10 in FIG. 1, as described below.

Applying the optical system 10 to a gaming console or gaming mobiledevice would allow for a significant increase in resolution of theimages (wave data) obtained by the gaming system due to the increasedamount of wave data (or information) retrieved from the environment 19.Specifically, the optical system 10 will allow the gaming system toobtain a holographic capture (“digital holograph”) of a whole scene inthe environment.

In order to obtain a digital holograph of an entire scene in anenvironment 19, the illumination of the optical system 10 must becontrolled. An exemplary embodiment of the optical system 10 for agaming device 36 is shown in FIG. 6. Note that the optical system 10 inFIG. 6 can be similarly applied to smartphones, cell phones, and otherstandalone devices with an illumination source 12 and a wirelessconnection to a processor 32. The illumination source 12 can be anylight source that emits spatial and temporally coherent light at eachlaser. For example, a single laser can be used or VCSEL array can beused where each laser in the array emits spatial and temporally coherentlight. As shown in FIG. 6, a continuous wavefront 18 (“illuminationwave”) is emitted from the laser 12 through the first diffractiveoptical element 12, rather than a discrete number of beams (as shown inFIG. 1). (In an alternative embodiment, the laser 12 can be replacedwith a VCSEL array 12, such as that shown in FIG. 5. In yet anotherembodiment, the laser 12 and the first diffractive optical element 14can be replaced with a VCSEL array 12, such as that shown in FIG. 5.)

This “illumination pattern” (i.e., continuous wavefront 18) blankets thearea of interest or scene in the environment without gaps, unlike aline/grid pattern or spot array. The laser (or other illuminationsource) 12 must provide enough illumination to blanket (i.e., cover) theentire scene in the environment 19 to be captured. Thus, instead of onlyreceiving information from reflected beams 22 from only where the lines(in the line/grid pattern) or dots (in the dot array) are projected, theblanket or cover of illumination receives reflected beams 22 from theentire scene.

The wave data 28 (e.g., phasorgrams) from the reflected beams 22 isreceived at a detector 26 and an image (digital holograph) 30 isproduced. The reflected beams (or return light) 22 reflected fromobjects 20 in the environment 19 are split in a similar way as shown inFIGS. 1 and 5, but the processing varies slightly because the algorithmwill attempt to judge distance across the entire field (environment 19),rather than just at the laser spots in an array or lines in a gridpattern.

The digital holograph 30 can be transmitted from the detector 26 to aprocessor 32 in a gaming console or mobile device 36 for formation of arepresentation 34 of the entire scene in the environment 19. The digitalholograph 30 can also be directly transmitted to a display (not shown)in the gaming console or mobile device 36.

In one example wherein the digital holograph 30 is transmitted to adisplay (not shown), participants viewing the display can interact withthe digital holograph 30. In addition, with this data, they couldinteract in mixed reality. This could include uploading (via theInternet) the digital holograph 30 to a social media platform, such asInstagram or YouTube, a mobile device application, such as FaceTime, oranother web application, such as one for telemedicine, a digitalteaching platform, a digital sales platform (e.g,. Amazon or Shopify),digital gaming platforms, digital advertising platforms, or any otherdigital platform. This very significant difference is a creation ofmixed reality that allows participants (or users) to interact with eachother and the virtual environment in a real-time in a three-dimensionalhigh-resolution manner, which is via a true interactive digitalholograph 30, or in a mixed reality environment. Such differences in thegaming system described above would allow the gaming system to moreaccurately process the environment and objects in the environment,thereby increasing the accuracy of the game.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

While various embodiments have been described and illustrated herein,those of ordinary skill in the art will readily envision a variety ofother means and/or structures for performing the function and/orobtaining the results and/or one or more of the advantages describedherein, and each of such variations and/or modifications is deemed to bewithin the scope of the embodiments described herein. More generally,those skilled in the art will readily appreciate that all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific embodiments describedherein. It is, therefore, to be understood that the foregoingembodiments are presented by way of example only and that, within thescope of the appended claims and equivalents thereto, embodiments may bepracticed otherwise than as specifically described and claimed.Embodiments of the present disclosure are directed to each individualfeature, system, article, material, kit, and/or method described herein.In addition, any combination of two or more such features, systems,articles, materials, kits, and/or methods, if such features, systems,articles, materials, kits, and/or methods are not mutually inconsistent,is included within the scope of the present disclosure.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as, “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (any formof contain, such as “contains” and “containing”) are open-ended linkingverbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements. Likewise, a stepof method or an element of a device that “comprises”, “has”, “includes”or “contains” one or more features possesses those one or more features,but is not limited to possessing only those one or more features.Furthermore, a device or structure that is configured in a certain wayis configured in at least that way, but may also be configured in waysthat are not listed.

The corresponding structures, materials, acts and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of one or more aspects of the invention and the practicalapplication, and to enable others of ordinary skill in the art tounderstand one or more aspects of the present invention for variousembodiments with various modifications as are suited to the particularuse contemplated.

What is claimed is:
 1. An optical system, comprising: a light sourceconfigured to emit a beam into an environment; a first diffractiveoptical element in alignment with the light source such that the beampasses through the first diffractive optical element, resulting in aplurality of beams emitted into the environment; a second diffractiveoptical element configured to receive one or more reflected beams;wherein the reflected beams are one or more of the plurality of beamsreflected back from the environment; a detector in alignment with thesecond diffractive optical element such that the detector receives theone or more reflected beams passed through the second diffractiveoptical element; wherein the detector is configured to determine wavedata from the one or more reflected beams and simultaneously generate aplurality of phasorgrams in a single image representing the wave data; aprocessor configured to receive the single image representing the wavedata and generate a representation of the environment; and a controlcomputer configured to receive the representation of the environmentfrom the processor.
 2. The system of claim 1, wherein the firstdiffractive optical element and the second diffractive optical elementare computer-generated hologram (CGH) devices.
 3. The system of claim 1,wherein the control computer is a computer within a vehicle.
 4. Thesystem of claim 3, wherein the control computer is configured togenerate a response for the vehicle based on the representation of theenvironment.
 5. The system of claim 1, wherein the light source is alaser.
 6. The system of claim 1, wherein the light source is a VCSELarray.
 7. An automated vision and detection system, comprising: animager configured to obtain a plurality of phasorgrams of anenvironment, each of the phasograms comprises phase information; whereinthe imager is configured to simultaneously generate a single image usingthe phase information; a processor configured to derive a representationof the environment based on the single image; and an implementationmodule configured to receive the representation of the environment andgenerate a response based thereon.
 8. The system of claim 7, wherein theimager comprises a light source configured to emit a beam into theenvironment.
 9. The system of claim 8, wherein the light source is alaser.
 10. The system of claim 8, wherein the light source is a VCSELarray.
 11. The system of claim 8, wherein the imager further comprises afirst diffractive optical element configured to receive the beam fromthe laser and split the beam into a plurality of beams.
 12. The systemof claim 11, wherein one or more of the plurality of beams are reflectedfrom the environment, resulting in one or more reflected beams.
 13. Thesystem of claim 12, wherein the imager further comprises a seconddiffractive optical element configured to receive the one or morereflected beams and split the one or more reflected beams, resulting inmodulated light.
 14. The system of claim 13, wherein the imager furthercomprises a detector configured to receive the modulated light from thesecond diffractive optical element and generate one or more phasorgramsof the modulated light.
 15. The system of claim 7, wherein theimplementation module is a module at a control computer of a vehicle.16. An optical system for generating digital images, comprising: a lightsource configured to emit a beam into an environment; a firstdiffractive optical element in alignment with the light source such thatthe beam passes through the first diffractive optical element, resultingin an illumination wave emitted into the environment; a seconddiffractive optical element configured to receive a reflectedillumination wave; wherein the reflected illumination wave is theillumination wave reflected back from the environment; a detector inalignment with the second diffractive optical element such that thedetector receives the reflected illumination wave passed through thesecond diffractive optical element; wherein the detector is configuredto determine wave data from the reflected illumination wave and generatean image representing the wave data; and a processor configured toreceive the image representing the wave data and generate arepresentation of the environment.
 17. The system of claim 17, furthercomprising a control computer configured to receive the representationof the environment from the processor, wherein the control computer is acomputer in at least one of a smartphone, mobile device, and gamingconsole.
 18. The system of claim 18, wherein the light source emitsspatial and temporally coherent light.
 19. The system of claim 17,wherein the processor is configured to transmit the representation ofthe environment to a web application.
 20. The system of claim 17,wherein the processor is connected to a smart phone.